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U. S. Office of Education Developmental Activities Program 
Final Report on Project No, X-005 

Project Title : Benefits of Research Design; a Pilot Study 

Project Directors Julian C. Stanley, Professor of Educational Psychology, 

University of Wisconsin (1404 Regent Street » Madison, 
53706) 

Dates of Project: 1 December 1964 - 31 August 1966 

Summary 

This summer program in the design and analysis of educational expert* 
ments was held at the University of Wisconsin in Madison during the eight* 
week period 21 June - 14 August 1965. The staff conalated of Richard E. 
Schuts and Julian C. Stanley, Co- Directors; Frank Baker, specialist in 
electronic computers and data processing; Cana V. Glass, instructor in sta- 
tist lea; and six graduate- student teaching assistants. There were 14 predoc- 
torsi and 16 postdoctoral participant*, ona of tham from Australia# Two 
pradoctoral and two postdoctoral participants wars woman# (The names and 
background information about the staff and participants appear in Appendix 
A of the final projact report.) 

Experimental daalgn, statistical analysis, and computar usage ware 
discussed each morning, Monday through Friday# Preparation of research pro- 
posals, special topics, and indapsndant study occupied the afternoons and 
Saturday mornings. Bssources of the University of Wisconsin, including its 
Laboratory of Experimental Design, ware availabla to each participant# 

Ona academic year aftar the close of the program, 29 of the 30 parti- 
cipants indicated on a questionnaire that they believed they seemed abler 
this academic ynev, because of tbs program, to design experiments, and 29 
fait Abler to analyse dete resulting from experiments# 
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p Sixteen of the 30 participants completed one or more experiments during 

the academic year 1965-66* Twenty-four of them "served as a consultant on 
the design and analysis of experiments" after the close of the program* To 
the question, "To what extent did the summer program seem to help you as such 
a consultant?," 0 responded "None," 0 responded "Little, " 7 responded "Mo- 
derately," 17 responded "A great deal," and 6 did not check any one of the 
four categories. 

Exactly half of the participants marked that they had "collaborated sub- 
stantially with others (either colleagues or students) on experiments since 
August 13" of 1965. Twenty- a lx said they had consulted on or collaborated 
on or pursued alone " research not strictly classifiable as controlled, 
variable-manipulating experimentation," and 21 responded "Yes (vs A "No") to 
the question, "If so, does it seem that your eight weeks in the summer prog- 
^ ram helped you do this better than you would have done otherwise?" 

To the question, Everything considered, how glad are you now that you 
spent the eight weeks in Madison?," the responses were as follows* 

0 I'm sorry I did. 2 Neutral: neither glad nor sad. 2 It was worthwhile. 

19 I benefited a great deal. 2 I* ”** * hugely rewarding experience. 

Other responses to the questionnaire are shewn In Appendix E of the 
final report. 

Though It la quite difficult indeed, especially only a year after the 
end of the program, to determine what effects the collaboration of the parti- 
cipants with the staff for eight weeks had, it would appear that the progress 

s 

of most of the predoc toral participants toward the doctorate has been 
facilitated and that the professional promise of a considerable number of 
the participants has been enhanced. Many more six- or eight-fieek summer in- 
> stitutes seem needed, particularly for professional personnel under the age 

of 40 who already have a doctorate. The areas of statistics, measurement, 

o 
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exper inert tal design, and data analysis seem to lend themselves weil to such 
treatment, although (as Lee J. Cronbach and others showed at Stanford Uni- 
versity during the summers of 1964 &~d 1965) institutes organised around 
actual empirical research are highly desirable, too* 
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Int redact ion 

Evaluating the effects of a summer program on the future careers of the 
participants is difficult. Persons who seek out such experiences are already 
likely to be energetic , successful achievers in later life. The postdoctoral 
ones have already embarked on careers 9 so each twig is bent in a rather defi- 
nite direction before the participant arrives for the program* The predoctoral 

* * 

participants are already committed to certain institutions and certain programs 
therein, and they will be preoccupied with doctoral requirements for one or 
more years following the program, thereby delaying the appearance of most evi- 
denc< „ of its impact on them. 

Eight weeks is a short time in one's life, so miracles of learning and 
redirection should not be expected from a program lasting only this long. Even 
if the program does have considerable cumulative influence on a person's career, 
this may be alow iu appearing and so interactive with other influences that it 
cannot be discerned clearly by the person himself or by others. 

Nevertheless, we must use whatever evidence that can be adduced to deter- 
mine whether or not auch programs are worth repeating and, if ao, how they 
should he modified in order to be more effective* Ideally, in the experimental- 
design sense, we should conduct the program as a controlled experiment, with 
a wall-matched control group that does not attend the institute, and follow up 
both groups for quite a few years in order to determine how they diverge. If 
recruiting begins early enough and the applicant group is able enough to provide 
both groups at a sufficiently high level, thii might be done, though the "reac- 
tivity" of tha disheartened rejectees, the self-fulfilling prophecy of the 
accepted persons, and the inability to control the sunmer activities of the 
rejectees, might undesirably affect the outcome of the experiment. Merely hav- 
ing on one's record the fact of attending a certain prestigious program, like 
displaying one's Phi Beta Kappa key, might be a powerful aid. We can hardly 
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falsify ths record (or award honor- society keys at random), so perhaps at best 
we could compare participation in two quite different types of summer programs 
of equal prestige, attended by persons assigned randomly to one or the other. 

Xn the present pioneering developmental effort we chose to take the 
seemingly best-qualified 30 persons from the applicants, thereby ruling out 
the possibility of having a meaningful control group* (See Appendixes A and B.) 
Our chief way of evaluating the success of the program is via reports from staff 
and participants, particularly the latter* In the progress report for the 
period 1 July - 30 September 1965 (see Appendix C) we summarize results from 
the questionnaire completed by the participants at the end of the program, on 
13 August 1965. The briefer progress report for tha 1 October 1965 - 30 June 
1965 (see Appendix 0) summarizes several further activities. 

Questionnaire Responses One Academic Year Later 
On 17 May 1966 we sent to each of the 30 participants the questionnaire 
shewn in Appendix S and, after several follow-ups of nonrespondents, received 
usable replies from all 30. Answers to the categorical questions showed that 
nearly all (29) of the participants felt themselves better able to design ex- 
periments and to analyse data resulting from them. Most of them (24) had served 
as a consultant on the design and analysis of experiments during the year, but 
only about half (16) hai "begun or completed any experiments since 13 August 
1965," Seventeen of tha 24 felt that the summer program had helped them as 
consultants "a great deal," and the other seven "moderately." 

Exactly half of the participants had collaborated with others on experi- 
ments since leaving the program. Twenty- six of them had engaged in "research 
not strictly classifiable as controlled, variable-manipulating experimenta- 
tion," and 21 checked that tha summer program helped them do this better 
than they would probably have done otherwise. 
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Only six have published or have "in press" anything affected, directly 
or indirectly, by the summer* program participation, *• but of course not enough 
time elapsed from 14 August 1965 until May or June of 1966 to allow most per* 
sons to complete an article, book, or monograph and have it accepted for pub* 
iication. Four had manuscripts arising partly from the program under consider- 
ation by editors. Nine had other completed reports that were affected by the 
summer program. 

Bight had secured research contracts or grants subsequent to 13 August 
1965, ranging in amount up to $605,000. Twelve had secured fellowships o* 
research assistantships for themselves or their students. A virtually rectangu- 
lar distribution resulted from responses to the four ordered categories of the 
question at the top of page 3 of the questionnaire: four persons felt that the 
summer program helped them "None" to secure the contracts, grants, fellowships, 
and/or research assistantships, five checked "Little," four checked Moderately," 
and five checked "A great deal." 

Six of 29 persons responding to the item indicated that their attendance 
at professional conventions seems to have increased, at least partly because 
of the program. Of the 13 who presented or coauthored papers at these convene 
tlons, five indicated that these papers were affected "None" by their activi- 
ties during the 1965 summer program, two checked "Little," four checked 
"Moderately," and two checked "A great deal." The summer program came too late 
to permit submitting papers for most fall conventions, such as that of the 
American Psychological Association. Convention activity of the participants 
seems likely to grew in future years, even though only four persons Indicated 
that they are "scheduled to appear on future convention programs." 

Of the 14 persons who taught courses with quantitative content during the 
12 checked that they had done so better because of the sunmer program - 



-7 



The overall evaluation of the program after one academic year was sought 
with the final item on the questionnaire: “Everything considered, how glad 

are you now that you spent the eight weeks in Madisonf 11 Ho one checked that 
he was sorry he- 'did* Two checked “Neutral: neither glad nor. sad* 1 ' Two checked 
"It was worthwhile*" Nineteen indicated that "I benefited a great deal*" 

Seven said "It was a hugely rewarding experience," 

Comments by respondents to the other items were Illuminating but so varied 
that categorisation is difficult* The majority who had attended any profes- 
sional convention had gone to the American Educational Research Association 
meetings in Chicago, 17-19 February 1966* The names of five appeared on that 
program, three of them twice* One participant presented a paper at the concur- 
rent convention of the National Council on Measurement in Education* Apparently 
the participants are not strongly affiliated with psychology, because only two 
of them had their names on the September 1965 convention program of the American 

Ik 

Psychological Association, and Just one on the September 1966 American Psycho- 
logical Association program* Only two of the participants, both of them pre- 
doc t oral, had academic backgrounds strictly in psychology, and both of these had 
already inclined toward educational research,* despite the fact that announce- 
ments concerning the program went to all members of the Divisions of Developmen- 
tal Psychology, Educational Psychology, and School Psychology of the American 
Psychological Association* Competition from Lee J. Cronbach's 1965 summer in- 
stitute at Stanford University, which was substantively rather than methodolo- 
gically oriented, may have been partly responsible for this, because his prog- 
ram may have appealed more to academic psychologists (who are usually reasonably 
well prepared In statistics and measurement) than did ours. 

A number of responses were received to the open-ended question, "How might 
the summer program have been conducted differently in order to help you better 
with the design and analysis of educational experimenter' Most of these 
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suggestions had appeared in the end-of- program evaluation in greater detail and 
had been summarized in the 1 July - 3C September 1965 progress report {see Ap- 
pendix C) . More emphasis on associations! studies and substantive research was 
called for. The persons less well prepared in statistics felt the need for a 
systematic set of lectures at a more intermediate level than ws required by 
most of the participants. More use of the participants' resources and skills 
should have been made. Some wanted the afternoon sessions more structured. 
Several postdoctoral participants objected to administration of the Doppelt 
Mathematical Reasoning Test, even though this was on a voluntary basis and only 
11 of the 16 postdoctoral participants chose to take it. The request for more 
time in which participants could interact informs 1 !y seemed to conflict with the 
requests for more formal structure, indicating that not all participants wanted 
the same approach. 

As might be expected, thus far the professional activity and productivity 
of the postdoctoral participants greatly exceeds that of the predoc toral ones, 
especially because nearly all of the latter went from the program right back in- 
to doctoral programs and became preoccupied with progressing toward the Ph.D. 
degree. Predoctoral participants were selected on the basis of test scores and 
academic records, whereas postdoctoral participants were «ot chosen directly in 
this way, so it seems likely that in the long run the predoctoral participants 
will become as effective educational experimenters as the postdoctoral partici- 
pants. In addition, the predoctoral participants now know quite a few nationally 
prominent educational researchers who can help them with their careers. 

The beneficial "fallout" of the program for the University of Wisconsin 
was, unintentionally, rather great. Three predoctoral participants shifted 
there, one in the fall of 1965 to continue her doctoral studies in counseling 
and guidance and two in the fall of 1966, one of these to be a postdoctoral fel- 
low in the Research and Development Center for Learning and Re-Education and the 
other to be an assistant professor of mathematics education. 
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One of the postdoctoral participants is spending the academic year 1966-67 
**■ ^ University of Illinois, working with one of the 1965 aummer-progvaa In- 
structors there. On* of the summer-program instructors is moving to the Univer- 
sity of Colorado in the fall of 1967 to work with a postdoctoral participant 
idiom he met at the summer program. One postdoctoral participant moved in the 
fall of 1966 to The Ohio State University to work with another postdoctoral 
participant there. Undoubtedly, other position changes at least partly attri- 
butable to the program have already occurred or will occur. All of these 
resulted outside the planned structure of the program. 

It seams likely that having on one's record the fact of attendance at the 
eight-week summer program in the design and analysis of educational exper imen ts 
at the laboratory of Experimental Design of the University of Wisconsin and, 
for several predoctoral participants, grades of "A" for course work there 
have already been considerable stimuli for promotions, salary increases, secur- 
ing of grants, etc., and will continue to be so in the future. Participants 
are likely to benefit professionally from being able to say that they have 
studied with (as staff of participants) a number of well-known educational re- 
searchers. They are more likely now to learn of opportunities for professional 
advancement, including positions, research projects and refresher courses. 

The initiative these participants showed by applying for the program and attend- 
ing it virtually guarantees that many of them will make distinctive professional 
contributions. A follow-up of the thirty persons in 1975 should be illuminating. 

Conclusions 

In general, it vould appear that this kind of institute should serve beat 
thsse able, young postdoctoral educational rsasarchsrs who did not hava axcal- 
lant preparation iu experimental deelgn, atatlatlce, and date proceeeing during 
their graduate work. It semis dsslreble not to Include person* who were exposed 
to strong methodological graduate programs, even though these did not seem to 
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"take." Also, persons whose publications indicate strong specialisation in non- 
experiaental areas such as as psychometrics are likely to be dissatisfied with 
s^ihasls on the design and analysis of controlled, variable-manipulating experi- 
ments and to press for greater attention to their favorite topics. Because not 

* . 0 

everything can be covered well in eight weeks, delimitation of material to be 
discussed seems essential. Getting participants before they become dedicated 
to a non- experimental methodology seems necessary, if the focus is to be on 
experimentation . 
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2* Abitrtet * 

m* Objectives : The purpose of thie project le to determine the benefits of 
e pilot research-design program that focuses on nee methods of experimental design 
and statistical analysis for educational research* The eight-meek session is 
meant to upgrade the research skills of 14 educational researchers mho already 
hold the Ph*D* or Bd*D* degree and to strengthen the preparation and aspirations 
of 14 predoctoral graduate students mho are preparing to do educational research* 
Emphasis mill be on the carefi l designing of controlled variable-manipulating 
experiments in the field of education* broadly defined* with analytical procedures 
considered as flowing from the experimental design* Outcomes of the project will 
be evaluated carefully* 

b* Procedures : The pilot project will be publicised widely via news- 
letters of professional organisations* professional Journals* and direct mall 
announcements* The 28 consultants should come from various areas of education 
and should have varying amounts of research experience* from much to little* 

The consultants will meet together each morning from 8:30 until 11:30* 

Monday through Friday* for lecture-discussion sessions on the following topics: 
principles and problems of experimentation under school conditions* measurement 
in experimentation (including multivariate analysis)* uses of electronic computers 
in experimentation* and the designing of research projects* From 1:00 until 4:00* 
Monday through Friday* they will work in 7 small mutual-interest groups* each 
composed of 2 predoctoral and 2 postdoctoral persons* Bach such group will have 
assigned to it for 4 hours daily a well-prepared advanced graduate student from 
the University of Wisconsin's Laboratory of Experimental Design who mill help the 
4 group members with technical problems of all sorts* including searches of the 
professional literature* Bach group will develop one or more substantial research 
proposals to be considered by everyone during the last several weeks of the 
seminar* It is expected that by the end of the summer a number of these pro- 
posals (perhaps as many as half of thorn) will be ready to submit to Federal 
agencies such as the U* S* Office of Education* The predoctoral consultants will 
meet on Saturdays* also* This eight-week summer program* the first of its kind* 
will be evaluated to determine the benefits that accrue to participants and to 
provide recommendations for subsequent research-training projects throughout the 
country* 

2 * Description of £he Activity . 

It is evident to persons who review research proposals for the U. S* Office 
of Education that nearly all projects have serious flaws in their experimental 
designs and analytical procedures* reflecting deficiencies in the preparation of 
their authors* Newer* more relevant techniques are seldom used* and older tech- 
niques are applied inappropriately* Confusion exists when interpreting associa- 
tlonal relationships* where both variables are free to vary without control by 
the experimenter, and functional relationships* where one variable is under the 
control of the experimenter and manipulated by him— roughly* the matter of corre- 
lation vs* "causation*" Comments by reviewers frequently are not understood by 
authors when relayed to them* because the latter lack the fundamental background 
In experimental design necessary to learn the new concept* For several years the 
Cooperative Research Branch coordinators have tried to upgrade the experimental- 
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design tophi* t lent ion of proposers. Their heroic efforts here not been wholly 
unsuccessful, hut It has become obvious to them end to referees of proposals that 
only better programs In universities across the country can improve the quality 
of the experimental designs and analyses proposed. 

Can an Intensive eight-week program for 28 able, well-motivated consultants, 
half of them postdoctoral and the other half predoctoral, help them learn newer 
techniques and become better researchers T Will they use their new knowledge 
effectively with their own doctoral students? It might be desirable to have one 
or more such seminars each summer for several years; the first one should serve 
as a pilot model, an evaluation device, and a generator of further interest. 

The pilot program would be a 40-day work session for the postdoctoral con- 
sultants and a 43-day work session for the others. In this amount of time with 
skilled staff It should be possible to Improve the experimental-design ability of 
the consultants considerably and to show them how they can continue learning on 
their own in subsequent years. Less than the 40 and 48 days of classes, dis- 
cussion, and small-group activity would seen too short a period In which to 
accomplish these two objectives. 

3. Related Literature . 

The proposed pilot project seems most closely related to three that the 
Social Science Research Council conducted about a decade ego* a pilot program on 
mathematics for social scientists at Dartmouth College during the summer of 1953 
and two such programs (at the University of Michigan and Stanford University) for 
nine weeks during the summer of 1955. The present Initiator was a postdoctoral 
fellow In the program at Ann Arbor, Where he learned a great deal about 
probability theory, set theory, matrix theory, and advanced calculus. 

A number of statistics departments, such as the ones at Iowa State University 
and Virginia Polytechnic Institute, offer regular summer-session Institutes In 
statistics, though not with emphasis on research In the social sciences. 

During the summer of 1964 at Stanford University, Professor Lae J* Cronbach 
conducted a large seminar concerned with research In teaching, its emphases seemed 
to be considerably more substantive than methodological. In the proposed pilot 
project the staff would emphasise principles of experimental design and analysis 
applied to whatever substantive Interests the participants bring with them and 
would assess the results of the eight weeks spent with the consultants. 

Lindquist (1953), Campbell & Stanley (1963), Stanley (1954, 1955, 1956, 1957A, 
1957B, 1958, 1960, 1961A, 1961B, 1961C, 1962, 1963A, 1963B, 1964A, and 1964B), 
Stanley A Beaman (1958), Pella et al. (1962), Stanley A Wiley (1962), Schuts, Page, 
A Stanley (1962), Page (1958), Travers (1964), and others have written about ex- 
perimentation under school conditions. Considerable discussion can also be found 
FR?C ^ ot * iar chapters of Cage (1963) and in Harris (I960). Pertinent sources less 
vLs directly related to educational experimentation are Edwards (1960), Ray (1960), 

I fifty flQSftV mr\A ftw it 0*17 \ it OO \ n...« /iao\ ^4 
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Fisher (1935). The Campbell-Stanley chapter and the Schuts-Page-Stanley 
curriculum guide should be good beginning materials for the lecture-discussion 
morning sessions of the institute* Most of the above materials , and others , 
will be used during the institute. Probably the books by Cox and Winer will be 
purchased by each participant and covered systematically. 
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4. Objectives . 

Immediate objective of the pilot project is to determine in whet ways 
and how effectively one in eight weeks can improve the competence of the 28 
consultante in designing and analyzing educational experiments, with special 
attention to principles and problems of experimentation under school conditions, 
measurement in experimentation (including multivariate analysis), uses of 
electronic computers in experimentation, and preparation of research proposals. 

An intermediate objective is to improve educational experimentation in the 
United States through the influence of the 28 consultants and the pioneering value 
of the pilot project as a method for upgrading research abilities rapidly. 



5. Procedure! . 
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Xh® pilot project would be held for eight weeks during the surjner of 1965, 
probably June 21 - August 13, at the University of Wisconsin, Madison, Wisconsin. 
Ttom 8:30-11:30 s.m. Monday through Friday the consultants would meet in a 
comfortable classroom for lectures by faculty members and discussion: . Approx- 
imately SOX of the 120 morning hours would be devoted to principles an d problems 
of experimentation under school conditions. About 25X would be devoted to 
measurement in experimentation, including multivariate analysis. The remaining 
25X would be devoted to computer technology (15X) and systematic consideration of 
the preparation of research proposals (10X). A main purpose of the project 

would be to determine whether these are the optimum percentages for such a 
program. 

The 120 afternoon hours, from 1:00 until 4:00 p.m. Monday through Friday, 
would be devoted to small-group work (4 participants and one assistant in each 
group), principally the preparation of research proposals for well-designed 
studies. Faculty members of the institute would consult with these 1 groups. 

In addition, the predoctoral consultants would meet on Saturdays. 

^ library of books, journals, and reprints would be set up from the Labor- 
atory of Experimental Design* a holdings for the convenience of participants to 
supplement the collections in the University of Wisconsin libraries, to which all 
participants would have access. 

The success of the institute would ba evaluated in several ways : 

1. By response of the participants, anonymously, to a questionnaire the 
last day and to another questionnaire 9-12 months later. 

2. By initial and final examinations covering the four principal lecture- 
discussion areas. 

3. By evaluation of U. S. Office of Education observers. 

4. By ratings of the faculty members and assistants of the institute. 

5. By an analysis of the quality of the research proposals produced, 
especially as reflected by the percentage of them that are submitted to a 
Federal agency and approved. 

6. By systematic comparison of the performance of predoctoral consultants 
with that of postdoctoral consultants. 

Steps in carrying out the proposal are as follows: 

1. 1 December 1964 — 31 March 1965. With the help of a half-time secretary 
and a half-time administrative assistant, publicise the pilot project widely. 
Prepare application forms. Carry on correspondence with professional associations 

journals, and prospective consultants. Final selection of consultants would be 
made by April 1. 
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2e 1 April 1965 - 31 May 1965. Half-time secretary and half-time adminis- 
trative assistant would make arrangements for the consultants, including housing, 
and would gather materials and equipment for the summer project, including books 
for the consultants and for the special collection of books, journals, and re- 
prints* The Initiator (JCS) would devote 50% of his time the second semester to 
studying the £opic o2 experimentation under school conditions and to consultation 
with the faculty members and assistants for the pilot project in order to set up 
a comprehensive syllabus to guide the lecture-discussion morning sessions and the 
propt> jal-des igning afternoon sessions. In mid-February, Professor Stanley will 
conduct a symposium on experimentation in educational settings at the annual con- 
vention of the American Educational Research Association in Chicago. 



3. 1 June 1965 - 31 August 1965. With the assistance of a full-time secre- 

tary, a full-time administrative, graduate assistants, consultants, and three 
other staff members, the initiator (JCS) would direct the pilot project, including 
preliminary arrangements and subsequent "clean-up" activities, 100% of his time. 



4. 1 September 1965 - 31 August 1966. The initiator (JCS) would prepare a 

report and evaluation of the pilot project, using 30% of his time for this pur- 
pose. The evaluation would involve all of the procedures mentioned earlier in 
this proposal. (Also, in November of 1965 Professor Stanley will stage a two- or 
three-day national seminar on experimentation in educational settings, sponsored 
by Phi Delta Kappa.) 

6. Personnel. 



Julian C. Stanley, Director of the laboratory of Experimental Design and 
Professor of Educational Psychology, University of Wisconsin, would direct the 
pilot project. Dr. Stanley has been a professor at the University of Wisconsin 
since 1957. He is a Fellow of the Divisions of Evaluation and Measurement, 
Educational Psychology, General Psychology, Teaching of Psychology, and Maturity 
and Old Age of the American Psychological Association, and a Fellow of the 
American Association for the Advancement of Science. He waj President of the 




National Council on Measurement in Education (1963-64) and the Tennessee 
Psychological Association (1951). He is a Director of the Psychometric Society 
(1962-65), a Representative to the APA Council from the Division of Evaluation and 
Measurement (1963-66), a member of the Research Advisory Council of the U. S. 

Office of Education, a member of the Committee of Examiners for the Aptitude Tests 
of the College Entrance Examination Board, a member of the Research Committee of 
Educational Testing Service, and a trustee of Test Research Service. He was 
Secretary-Treasurer of the Division of Educational Psychology of the American 
Psychological Association (1960-63), which he now serves as President-Elect. He 
Is also Vice-President of the American Educational Research Association, chairing 
the Division of Measurement and Research Methodology. Dr. Stanley is a member of 
numerous other professional organizations, including the American Statistical 
Association and the Institute of Mathematical Statistics. He is a consulting or 
advisory editor for the Journal of Educational Psychology, the American Educational 
Research Journal, and the Journal of Educational Measurement . He is the author of 
125 artldea, notes, and reviews and of a textbook. Measurement in Today *8 Schools 






ed ^ t , t ^Z , 19 ^)* 8e*tdes earning i doctorate In educational 
vaniev Se ^ “»«^**d |Jnlveralty (February 1950), !ie ha* studied at the Uni- 
S2JSi« i.! w * 8k !» Bwatt ot *955), the Department of 

of n«iS.J? , rt e ?K t ^> f A Chl S 880 <U ® ow:h8 » 1955-56), and the University 

if l8 rT (8“16rlght Act Research Scholar, 1958-59). He has taught at 

*5 ! lty of Geor8l* (*wa»er of 1967), Harvard University (1968-49 and sumoar 

cLita ri9frz.?*^ 0dy S° l 3 e8 .Vf or T “ ch « 8 <1969-53). the diversity of Wi!T 

to°the Atlmta U ? lw * r8 * t y of Hawalt < 8un,ner o£ I960), In addition 

Z n * ™* nt * (Georgia) high schools (science and mathematical 1937-42) and 

!KS f uni ? r c °ll*8 e (psychology, part-time, 1966-68). Ha Is 

arl ^DerlS^i^r.Tro 422 ^ *" d ^S£i£gB Men M Sci e nc e . His special Interests 

diii^n tnd iMASurtiDdnc • 

«f ..? fn .iWr*- d \* ® cbut *» Professor Of Educational Psychology at the University 
° n * Otenp' e), has agreed to serve as associate director of the pilot 

,T , a r i.. H r 8 * w ll0H .° £ J he Amerltan Psychological Association and the Editor 
r.i Hucgtional Heasugenent. A graduate of the University of 

«J"f . *5j Lo ’. An ®f 1 ^ 8 * ma -) oring ln history and being elected to Phi Beta 

rX / i, „ ? _ 8 Ph ,' t) ’ d ® gree in educational measurement from Teachers College, 

fell^ihl^ 1 ^ 4 c-h » A V d0C 5?!f a > student he held the AERA-World Book Company 
fellowship. Dr. Schutt has published a considerable number of articles In pro- 

C!!! !!! a L JO H!? a1 **. Dur i n * the »w»“er of 1962 he and Professor Ellis B. Page 
^sr w- n^ V «!:* 1 J y ° f wl “ co "sin, where they and Professor Stanley developed 

r » r !r r h 8 T? Ie T cu ! rlc “ lum «" ld « ** ineervlce training In educational 

” ,<a ” h w " mentioned under Category 3, above. His special Interests are 
educational research and measurement. Including factor analysis. 

... ® e * yr *I* t . 1 ®' 8®hcr, Associate Professor of Educational Psychology at the 

the V nrai.ei- 0f ha f. ? greBd to be the electronic-computer specUlist for 

the project. Dr. Baker took his Ph.D. degree ln educational psychology at the 

Unlvers Ity of Minnesota (1959), vhere he had considerable experience with the 

* 1504 digital computer while it was being developed. 

- tb * di 8 l tal-computer consultant for the social sciences at the 

«!? °f W1 ? c ™®in. E * ch semester he teaches a course entitled "Computers 

ll 2 n fti^nilf» l |^ Cle “ Ce8 ‘ Throu8h ht8 swmerous publications and consulting, 
n« is nationally known among computer specialists# e 

at thf^lvers^v’J^-f T , itl \ IV / ell ? w J ln the “sslgn of Educational Experiments 
^ Z* Wl8c °“« ln "ho is scheduled to receive his Ph.D. degree In or 

oar^in^MM^L^ 88 88reed to be 8,4 instructor In the program. He Is well pre- 
* 8ti *J lcs 8nd measurement and experienced as an experimental design con- 
r ba ® studied multivariate analysis with Professors Chester W. 
Har.is and Henry F. Kaiser and measurement theory with Dr. Frederic M. lord. He 
la the author of several published professional papers. 

The administrative assistant for Professor Stanley and the seven graduate 

the small groups would be chosen from among advanced 
“ . ln . 5 h . e . hsboratory of Experimental Design, of whom there will be 

at least a dosen available. These persons will be doctoral candidates beyond the 
master s degree, who ln most instances will have already completed at least two 
years of course work with specialisation In statistics and measurement. 
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Besides tne laboratory of Experimental Design end the Wisconsin Center, the 
consultants would have access to the Statistical Laboratory, the Numerical 
Analysis laboratory with its digital computers, the Survey Research Center, the 
laboratory for Research on learning and Instruction, the Psycho-Educational Clinic, 
the Reading Clinic, and two laboratory schools, in addition, the State Department 
of Public Instruction is a few blocks away, 

8* g&tts Mroitte* 

(a) There is no support available for this project from sources other 
than the University of Wisconsin, 

(b) This proposal has not been submitted to any other agency or organization. 
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Mama 




Pradoctoral Melee 
Pegreee 


^itle— Place of Employment 


Barry i* Bowman 


B.Th. 

B.A. 

Me A • 


1959 Pentecostal Bible Inst a 
1962 Union University 
1964 George Peabody College 
for Teachers 


HDRA Pel low* -George Peabody 
College for Teachers 


Jersey D. Finn 


B.A. 


1963 Syracuse University 


HDSA Fellow— University of 
Chicago 


Andrew R. Conyee 


B*A» 


1964 College of the Holy 
Cross 


Graduate student— Clark 
university 


Bobby X, Hopkins 


B.A. 

M.S. 


1954 Pasadena College 
1959 University of Southern 
California 


Assistant Professor of 
Mathematics and Science— 
Blola College, La Mirada, 
California 


John H. Jonklno 


BaBd. 

H.Bd. 


1955 University of Miami 
1961 University of Miami 


Principal— Miami Springs 
8enior High School 


tool* G. King 


BaAa 

N.Bd. 


1951 Antioch College 
1959 Wayne State University 


Director of Laboratory 8chool 
and Associate Professor of 
Education— Antioch College 


Danlol Langncyer 


B.8. 


1963 Brooklyn College 


Research Assistant, Center 
for Advanced 8tudy of Educa- 
tional Administration— Univer 
aity of Oregon, Eugene 


Sosa W. Millar 


B.8. 

K.3. 


1952 University of Georgia 
1964 Auburn univarsity 


Graduate Assistant in Educa- 
tion and Administration, 
University of Mew Mexico^ and 
Assistant Profsssor of Educa- 
tion, West Georgia College 
(on leave) 


Casper F. Paul son, B. A. 


1949 Augustana College 


Bvolvlng Programs Director- 


iTa 


M.A.T 


,,1961 Reed College 


Teaching Research Division, 
Oregon College of Education 


Thomas A. Ronberg 


B*Sa 

M.S. 


1955 Omaha University 
1959 Omaha University 


Coordinator of Research Test- 
ing Development— 8chool of 
Mathematics Study Group, 
Stanford University 


Richard S. Steudt 


B.8.R 

MaA. 


• 1956 University of Michigan 
1963 University of Michigan 


HDRA Fellow— University of 
Michigan 
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(Pjtsif L. Wardro p 



Title— Place of Employment 



B • A i 



1963 Washington University NSF Fellow— 'Washington Uni- 
versity, it* Louis 



I t eMeM i to k 

Predoc toral Females 



Carole A. Congram 



B.A. 1958 Tufts University 
M*Bd • 1965 University of Worth 
Dakota 



Graduate Assistant in Coun- 
seling and Guidance— Univer- 
sity of North Dakota 



Carrell P* Horton 
(Stipend provided 
by Fisk Univer- 
sity) 



D*A* 1949 Fisk University 
M*A* 1950 Cornell University 



Statistician, Department of 
Pediatrics— Heharry Medical 
College, Nashville, Tennessee 



Postdoctoral Males 



Glenn C* Boerrlgter 
(Part 1c ipant- obser- 
ver from U*S* Of- 
fice of Education) 



B.A. 1953 Nebraska Wesleyan 
University 

M.Ed. 1957 University of Nebraska United States Office of 
Kd.D. I960 University of Nebraska Education, Washington, D.C. 



Research Coordinator- 
Cooperative Research Branch, 



Dale L. Bolton 



B.S. 1950 Oklahoma 8tate Uni- Assistant Professor of Educe- 

varsity tlonal Administration— 

M.S. 1953 Oklahoma 8tate Uni- University of Washington, 

varsity Seattle 

Ph.D. 1958 University of Wisconsin 



Donald Fitzgerald 



A.8.T.C* 1953 Sydney Technical Senior lecturer in Education- 

College al Research— University of 

Ac*Dlp.Ed. 1959 London Univer- Melbourne, Australia 

slty 





M.A. 


1960 




Ph.D. 


1962 


D* Bruce Gardner 


BeSe 


1948 




M.S. 


1949 




Fh.D. 


1952 


Burton 1* Grover 


B.S. 


1954 




M.A. 


1960 




Ph.D. 


1963 


Kenneth D. Hopkins 


B.A. 


1956 




M.S. 


1957 




Ph.D. 


1961 



isadena College * Associate Professor of Educe- 
ilverslty of tional Psychology— University 
8outhem California of Southern California 






Southern California 






ERiC 
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Degrees 


Title* Place of Employment 


^ John L. Horn 


B.A. 1956 University of Denver 
M.A. 1961 University of Illinois 
Ph*D. 1965 University of Illinois 


Assistant Professor of 
Psychology- -University of 
Denver 


Jicob Me lCigan 


B.A. Brooklyn Collage 

II. A. City College of New York 

Plt.D. 1965 New York University 


Instructor in Educational 
Psychology— Brooklyn Col- 
lege 


Jason Millaan 


B.S. 1955 University of Michigan 
M.S. 1958 University of Michigan 
Ph.D. 1960 University of Michigan 


Assistant Professor of 
Educational Psychology— 
Cornell University 


Daniel P. Norton 


B.S. 1949 University of Minnesota 
H.A. 1958 University of Minnesota 
Ph.D. 1964 University of Minnesota 


Assistant Professor of 
Education-- Indiana State 
College 


Daniel L. Stufflebaaa 


B.A. 1958 University of Iowa 
M.S. 1961 Purdue University 
Ph.D. 1963 Purdue University 


Director of Test Develop- 
ment Center and Assistant 
Professor of Education— 
Ohio State University 


Donald A. Trismen 


B.S. 1951 State University of New 
York at Fredonla 

M.M. 1952 University of Rochester 
Ph.D. 1964 Cornell University 


Associate for Research and 
Development- -Educational 
Testing Service, Princeton, 
New Jersey 


Paul A. Twelker 

• 


B.A. 1958 San Diego State College 
M.A. 1963 University of California, 

Los Angeles 

Rd.D. 1964 University of California, 

Los Angeles 


Assistant Research Profes- 
, sor— Teaching Research 
Division, Oregon State 
, System of Higher Educa- 
tion, Monmouth 


Kaoru Yamamoto 


B.S. 1953 University of Tokyo, 

Japan 

M.A. 1960 University of Minnesota 
Ph.D. 1962 University of Minnesota 


Assistant Professor of 
Educational Psychology— 
University of Iowa 
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Postdoctoral Females 
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o 

ERJC 

MflWtBffllMI 



Carmen J. Finley 


B.A. 


196 S 




M.A. 


1952 




Ph.D. 


1962 


Mary C. Regan 


B.S. 


1952 




M.S. 


1959 




Ph.D. 


1963 



bi 



University 
achers College* 
bia University 



Data Processing* -Sonoma 
County Schools , Santa 
Rosa, California 



University of California, 
Davis 
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Hatt^ 



Frank B. Baker 



Gene V* Glass 



Richard R. Schutz 



Julian C. Stanley 



B.S. 

M.A. 

Ph.D. 

B.A. 

M.S. 

Ph.D. 

B.A. 

M.A. 

Ph.D. 

B.S. 

Bd.M. 

Ed.D. 



Alan M. Abrams 



B.A. 

M.S. 



Elizabeth A. Holstein B.S. 

H.A. 

Tom R. Houston B.A. 

M.A. 

Dorothy L. Jones B.S. 

M.Ed. 

Eduard P. Meyer B.S. 



Theresa Penna Firms B.S. 

M.S. 



Degrees 

Faculty 

1950 University of Minnesota 

1954 University of Minnesota 
1959 University of Minnesota 

1962 University of Nebraska 

1963 University of Wisconsin 
1965 University of Wisconsin 

1951 University of California 

Los Angeles 

1955 University of California 

Los Angeles 

1957 Columbia University 

1937 Georgia Teachers College 
1946 Harvard University 
1950 Harvard University 

Assistants 

1964 Yale University 

1965 University of Wisconsin 



1957 University of Wisconsin 
1963 University of Wisconsin 



1964 Harvard University 

1965 University of Wisconsin 



1951 West Chester State 
College 

1961 Pennsylvania State 
University 

1965 University of Wisconsin 



1962 Catholic University of 
Rio de Janeiro, Brasil 

1963 University of Wisconsin 



Title-Place of Employment 



Associate Professor of 
Educational Psychology- 
University of Wisconsin 

Assistant Professor of 
Education— University of 
Illinois, Urbana 

Professor of Educational 
Psychology— Arizona State 
University 



Professor of Educational 
Psychology— University of 
Wisconsin 



Graduate Assistant-* 
Laboratory of Experimental 
Design, University of 
Wisconsin 

Administrative Assistant- 
Laboratory of Experimental 
Design, University of 
Wisconsin 

NDEA Fellow— Laboratory of 
Experimental Design, Uni- 
versity of Wisconsin 

Graduate Assistant— 
Laboratory of Experimental 
Design, University of 
Wisconsin 

Graduate Assistant- 
Laboratory of Experimental 
Design, University of 
Wisconsin 

Graduate Assistant- 
Laboratory of Experimental 
Design, University of 
Wisconsin 
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Andrew C* Barter 



Degrees Title— Piece of Employment 

B*S. 1963 Indiene State Univereity Program Coordinator— 

M.S. 1965 Univereity of Wleconeln Leboretory of Experimental 

Design, Univereity of 
Wleconeln 



Adrien Ven Handfrene 



B.S. 1963 Univereity of Utah 
M.A. 1964 Univereity of Utah 



Creduate Assistant— 
Laboratory of Experimental 
Design, Univereity of 
Wleconeln 



Secretary 



Typist and Clerical Aeeletant 



Harriet Clutterbuck— Wisconsin Civil Steven R. Cox— Senior, University of 

Service Stenographer XII Wisconsin, majoring in mathematical 

economics 











APPENDIX C 



Progress Report on 

U. 8* Office of Education Cooperative Research 
Developmental Activities Program 
Project No* X-005 

for the period July 1, 1965 through September 30, 1965 

Name of Institutions University of Wisconsin (Madison) 

Title of Project : Benefits of Research Design; A Pilot Study 

Project Director s Julian C* Stanley, Professor of Educational Psychology, 

University of Wisconsin 

The eight-week summer program in experimental design at the University of 
Wisconsin continued, with all thirty pre- and post-doctoral participants parti- 
cipating regularly, through 13 August 1965* The four instructors and seven 
graduate assistants worked regularly with the group, formally and informally, 
to maximise the benefits the Individuals received from their eight weeks of 

Also, all the participants attended the 2 1/2-day Seventh An- 
nual Phi Delta Kappa Symposium on Educational Research at the University of 
Wisconsin, 9-11 August 1965* 



During the afternoon of 12 August 1965, the participants prepared indivi- 
dual evaluations of the program, as they saw it near the end of the eighth week. 
These were read carefully by Professors Schuta and Stanley, who served as as- 
sociate program director and program director, respectively, and by Professors 
Baker and Glass, the other two instructors. Partly on the basis of the Impres- 
sions gained from perusing these structured essays. Professor Stanley reports 
the following tentative evaluation, subject to modification when evaluation by 
structured questionnaires takes place some six months hence. 



The typical respondent reported that he felt the eight weeks* participa- 
tion was well worth the effort ha made to come to Madison and participate in 

J 0n * P erton cmo * •uch specialised psychometric interests that 

he did not welcome the strong emphasis on the design of controlled, variable- 
manipulating experiments. One or two others wished there had been more emphasis 
on substantive research and leas on methodology* The typical participant worked 
hard, often studying far into the night* Even when his prior expectations did 

with the organisation of the program, he was able (because of 
flexibility in himself and, to some extent, in the program) to accomplish a 
great deal that he considered beneficial. 



About five of the participants had not really had a suitable course in 
statistical inference and were, therefore, handicapped in this area. Probably 
it would have been wise to have tested doubtful applicants with respect to 
statistical knowledge before admitting them, or to have had two levels of 
statistical Instruction. (The participants were offered the opportunity to 
form a more elementary lecture-discussion session, but they preferred to remain 
with Dr* Glass, Who proved to be a well organised, stimulating teacher.) 

In addition to statistical background, it would have been desirable to 
have screened the group better for quantitative aptitude, because fast learning 
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ability it essential in such a short session* For this purpose we recommend 
tha Dopnelt Mathematical Reason ins Test (W®C of The Psychological Corporation, 
304 East 43th Street, Maw York 17, New York, a one-hour test available only at 
a number of testing centers throughout the country that offer the Miller 
Analogies Teat* Two of the 14 predoctoral participants had offered HURT scores 
upon applying for the program, and 23 others agreed to take the tent during the 
summer. (Five postdoctoral participants preferred not to do so, and were not 
required to take tha test. At least three of these five seemed highly able 
quantitatively.) The 25 DMRT scores ranged from 21 points (2 persons) to 48 
points (3 persons) out of a possible 50 points, tha median being 34. The thr*e 
top scorers were all predoctoral; and in general the 14 predoctoral partici- 
pants, all of whom took the DMRT, scored slightly better (median, 36) than did 
the 11 post-doctoral participants who chose to take the test (median, 34). The 
difference is most noticeable in the upper fifth of the 25 persons, all 5 of 
whom were predoctoral. A score of even 21 on the DMRT is not low for graduate 
students in education, however, being the 60th Tile of that group. The 48*0 
are extremely high, being in approximately the upper IX of graduate students in 
statistics departments. Probably a score of at least 30 is desirable for ef- 
ficient learning in such a group (other aspects being favorable), but several 
persons who scored lower than that seemed to be among the participants who 
benefited most. 



The combination of 14 predoctoral and 16 postdoctoral participants (origi- 
nally exactly 15-15, but one of the former completed hie degree after being ac- 
cepted as predoctoral and continued to be paid in that category) seemed 
desirable to everyone, except that it seams unwise to choose predoctoral appli- 
cants who have completed less than two years of full-time graduate study and 
prepared a thesis. Persons with less than that may not be well enough motivated 
to work effectively on their own a great deal of the time, or greatly Interested 
In research. 

The age range, from 22 to 40, provided desirable heterogeneity of 
experiences, as did the varied backgrounds of the participants, from high-school 
principal in Miami to full professor of child development in Iowa. High general 
intelligence, as reflected by a score of 70 or more on the Miller Analogies 
Test (MAT) or the equivalent (say, the 90th tile) on the Graduate Record Ex- 
amination, Verbal-Aptitude Test, seems desirable. Most of the participants 
seemed to be above or not too far below this level, but because no such measures 
are available for post-doctoral participants (whereas all pre-doctoral partici- 
pants except one had to furnish such Information when applying), we cannot be 
sure. The 10 MAT scores provided by the 14 pre-doctoral participants were 90, 
89, 85 , 79, 76, 72 , 70, 70, 61, and 58, for which the median is 74, but the 
four who chose not to have the MAT information sent may have known that their 
scores were low. 



The 8 GRB-V scores provided by the (originally) 15 predoctoral persons 
were 820, 750, 690, 670, 630, 590, 560, and 530, plus one pre-doctoral parti- 
cipant for whom the GRE-V was reported merely as "99th Xlle," probably some- 
where in 700's, which would make the median 670, the 94th Xile of GRB-V norms. 
The lowest score, 530, is the 65th Xile. 



The range of the nine GRR-Quantltative Aptitude scores was from "76th Xile" 
to 820; the median was 650. The 6 GRE special-field scores ranged from 500 
(47th Xile of history majors) to two 680*0 (education and psychology, respec- 
tively), with a median of 610. 
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On« p re- doctoral participant acored 90 on the MAT and 48 on the DMRT, but 
vu* scored only 58 on the MAT scored 48 on the DMRT (and 820 on the GRE-Q). 
The third person who scored 48 on the DMRT scored 79 on the MAT, 820 on the 
GRE-V, but only 650 on the GRE«Q. These discrepancies suggest that it will be 
unwise to use only a single test score in a cut-off fashion for selecting par- 
ticipants for summer programs. It would seem desirable to have four scores 
for each applicant: MAT, GRE-V, DMRT, and GRE-Q. Because it is difficult to 
obtain all theta lor pre- doctoral applicants * and virtually Impossible to ae- 
cure unbiased such scores for post-doctoral applicants, many of whom are in 
charge of administering such tests, the selection committee for a program must 
begin its efforts early. If told in time, most pre-doc toral applicants can 
arrange to take all four tests (at Just two administrations, MAT and DMRT at 
one and GRE-V and GRE-Q at the other). Post-doctoral applicants can be urged 
to nave sent to the selection coonittee the scores from such tests that they 
took as graduate students f and preference can be given to applicants Who offer 
at least some of the scores. Most graduate students take either the MAT or the 
ORE, and many take both. The DMRT is much less widely used. 

The else of the group (30 participants) seemed right to most of the 
respondents, but several of them indicated that mere individuation via smaller 
subgroups might be desirable. A few thought that 45-50 persons could be 

lectured as well as 30, if sufficient staff for smaller discussion groups were 
available. 



Generally, neither the participants nor the Instructors knew quite how to 
utilize the services of the graduate assistants well enough. One of these 
taught matrix algebra effectively to a small group of interested persons. 
Another helped with computer programming. A third conducted some tutorial ses- 
sions for 3-5 persons whose statistical training had beon too slight. However, 
it seems better to have not more than one 20 hour- per week graduate assistant 
for each 10 participants and to use the freed funds to hire more regular, full- 
staff. The graduate assistants used should not be preoccupied with urgent 
academic matters of their own, such as theses and preliminary examinations for 
degrees. It might be well to import them from another university to forestall 
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Jhe director. Professor Stanley, was plagued by too many local responsi- 
bilities that he could not delegate or relinquish, such as eight M.S. degree 
theses and the accompanying oral examinations, planning for the Phi Delta Kappa 
research symposium, and getting ready to move to California for a year at the 
Center for Advanced Study in the Behavioral Sciences. Dr. Stanley handled a 
75-minute session on the design of experiments each morning, five days a week, 
for seven weeks. Along with his other duties, this almost worked him to death, 
trying to prepare materials for the sessions. He recommends that the organiser 
and director of a program free himself from as many other commitments as pos- 
sible and serve as director and stimulator only (i.e., not be a formally com- 
mitted instructor). 

Perhaps the program should have lasted seven weeks, instead of eight, be- 
cause both staff and participants seemed to run out of steam near the end. 

This was alleviated partially by the two-and- one- half-day Seventh Annual Phi 
Delta Kappa Symposium on Educational Research, organized by Professor Stanley 
and held on the University of Wisconsin campus 9-11 August 1965. The summer- 
session participants attended the five main speeches (by Julian C. Stanley, 
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George E. P. Box, Ingram Olkin, Leslie D. McLean, and Frank B. Baker) and heard 
tape racordings of the closed discussion sessions that followed each of the 
speeches. Also they received copies of most of the speeches. For the upper 
fourth of the participants, six week# might have been long enough. For the 
lower fourth, eight or more might have been needed. 

^e stipends ($100 per week) and per-diem allowances ($80 per week) for 
post-doctoral participants seemed adequate to them, but the lack of stipends 
for pre-doctoral participants (who received Just $96 per week per diem) was 
unfortunate. There seem to be good reasons why both pre- and post-doctoral 
participants should receive the same remuneration (about $200 per week plus 
travel allowance seems desirable), especially because the former usually have 
no funds for the summer other than those paid for the program, whereas post- 
doctoral participants have been paid reasonably well by their institutions for 
wie preceding academic year and may be having their summer stipends supplemented. 
Tne pre-doctoral participants functioned as well as consultants as did the post- 
doctoral ones. Dependency allowances might be desirable, also, for programs 
lasting six weeks or longer. 

The participants made a number of suggestions for future programs of this 
and other kinds. They prized time for private study and reflection, and ready 

var ^ staff members and consultants. More outside consultants 
would have been desirable this summer. The participants were encouraged to re- 
quest these by name during the program, and those requested were brought in. 

Process works too inefficiently to rely on exclusively, so consultants 
should be scheduled in the designing of the program Itself. 

The staff was not highly successful in utilizing special research conmeten- 
cles of the participants themselves, perhaps largely because of the restriction 
of the program to the design and analysis of experiments , an area in which most 
of the participants had not yet contributed greatly. One pre-doctoral partici- 
pant, a specialist in the computer programming of multivariate analysis of 
variance, worked with certain other participants on this. Probably it would 

have been possible to utilize the special competencies of several others bet- 
ter than was done. 

Several participants who were greatly interested in psychometrics tried 
continually to enlarge the nature of the program from consideration only of 
true and quasi- experiments to encompass associatlonal studies, but the 
staff resisted this pressure, because it was felt that the primary need of the 
group during this particular eight-week session was for the former. Partici- 
pants had been selected because, although they might already have contributed 
considerably to associatlonal studies, they were less well prepared to do 
experiments • 

All in all, this pioneer program in the design and analysis of educational 
experiments went about as well as could be hoped for. The staff feels that 
nearly all of the 30 participants will be better educational researchers than 
they otherwise would have been. Specific changes at least partially attribut- 
able to the program will be sought from participants in about six months. 

One of the ablest, most perceptive of the postdoctoral participants ended 
his evaluation as follows, reminding us that miracles should not be expected— 
certainly, not quickly- -from one eight-week session: 
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'*• • * program did one thing, namely, foreing me to wonder 
About many things, and this was good* Someone must blare a trail, 
and this program did it, with all the understandable shortcomings 
and ambiguities* In the £inal analysis, we should not be depend* 
ing on the grandpas (pardon the expression) who organised the 
program, precisely because the future depends upon utiat we do, not 
upon what they do* Thus, we will keep plodding our royal road 
of learning. 

"Well done pioneers!" 



(This man was just three years younger than the average of the four in* 

8 true tors for the program** the "grandpas" to whom he referred**and only 14 
years younger than the oldest Instructor* Either the program acted on him 
like Ponce de Leon's fountain of youth, or else the instructors looked pre- 
maturely aged by the time it ended.) 

Changes Made During This Reporting Period 

Attention is invited to Part 5, "Procedures," of Appendix A for Project 
No. X-005* The success of the program was evaluated in ways la (but not 
anonymously), 3, 4, and 6* It did not prove feasible to give initial and 
final examinations to all the participants, because of their heterogeneity of 
backgrounds and heterogeneity of interests. It was felt that trying to keep 
all participants in a common curriculum would be educationally unsound. 

Step No. 5, "By an analysis of the quality of the research proposal pro- 
duced, especially as reflected by the percentage of them that are submitted 
to a Federal agency and approved," will be carried out in the future. 



Julian C. Stanley 
Project Director 
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APPENDIX D 

* 

Progress Report on 

U. S. Office of Education Cooperative Research 
Developmental Activities Program 
Project No. X-005 

for the period 1 October 1965 through 30 June 1966 

Name of Institution: University of Wisconsin (Madison) 

Title of Project : Benefits of Research Design; A Pilot Study 

Project Director : Julian C. Stanley, Professor of Educational Psychology, 

University of Wisconsin 

During the period of this report Professor Stanley and his half-time 
administrative assistant. Miss Elizabeth Holstein, have kept in touch with the 
30 persons who participated in the summer 1965 program at the University of 
Wisconsin and with the instructors (Baker, Glass, and Schutz) and assistants 
who helped conduct it. They have been sent various materials to refresh and 
supplement the eight-week experience. Journals and convention programs have 
been searched for their contributions. An all-day set of speeches and con- 
ferences on the design and analysis of educational experiments was held speci- 
fically for this group on 16 February 1966 at Loyola University in Chicago, 
with good attendance. This was planned by Professor Stanley and Miss Holstein, 
with Professor Samuel T. Mayo of Loyola University as the chairman. Five 
major addresses were given by outstanding specialists, and then small-group 
sessions were held. 

In mid-May of 1966 a questionnaire concerning the first-year effects of 
the summer program was sent to each of the thirty former participants. Replies 
•re beginning to arrive. These will be tabulated and analyzed as a basis for 
preparing the final report at the end of the summer . 

Professor Stanley and Miss Holstein maintain a considerable correspondence 
with the participants. Preliminary evidence indicates that their progress has 
been facilitated considerably by the summer experience, this being at least 
as true of the predoctoral participants as of the postdoctoral ones. 

Direct and indirect influences on the instructional staff seem to have 
been great, too. Richard E. Schutz, now Director of the Southwest Regional 
Laboratory for Educational Research and Development, conducted an AERA- sponsored 
five-day session on experimental design and analysis in Chicago 12-16 February 
1966, immediately preceding the annual conventions of the American Educational 
Research Association and the National Council on Measurement in Education. 

His faculty consisted of Gene V. Glass, Leslie D. McLean (who had been a one- 
day consultant for the summer program), and Julian C. Stanley. Fifty persons, 
most of them already having doctorates, attended from all over the United 
States . 

During the period 11-15 February 1967 Gene V. Glass, now an assistant 
professor at the University of Illinois, will conduct a five-day AERA presession 

on the design and analysis of experiments at Gros singer. New York,, for some 50 
persons . 
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APPENDIX E 



The One Academic* Year Follow-Up 

1* The three letters of transmittal. 

2. Sunmary sheet. 

3. The questionnaire responses 

a. Predoc tors l participants 

b. Postdoctoral participants 
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CENTBR FOR ADVANCED STUDY IN THE BEHAVIORAL SCIENCES 
202 Junipero Serra Boulevard - Stanford, California 94305 



Telephone (415) 321-2052 



Hay 17, 1966 

SUBJECT: Your evaluation of the 1965 Summer Program In Experimental Design, 

nine months later 

TO : All thirty persons who were participants In the 1965 Summer Program 
In Experimental Design at the University of Wisconsin, 21 June - 
13 August 

FROM : Julian Stanley 

Dear Thirty: 

You've had an academic year in which to think about the aftereffects of 
the eight weeks you spent in Madison last summer. The U.S. Office of Educa- 
tion (which financed the sunnier program), Frank Baker, Gene Glass, Dick Schuts, 
and I would greatly appreciate your frank comments, criticisms, and sugges- 
tions concerning the program. These will be used by us in preparing a final 
report for the USOE, in whi~h no commenter's identity will be divulged. On 
the enclosed sheets I have tried to provide some structure for your evaluation, 
but please feel free to go beyond this and remark about whatever aspects you 
wish. Do the best you can to estimate the extent to which your various acti- 
vities during the summer actually influenced your subsequent achievements and 
activities. Please try to minimise the "post hoc, ergo propter hoc" ("after 
this, therefore because of it") fallacy. 

I have noted, for example, that four of you (Bowman, Mil lawn, Twelker, 
and Yamamoto) presented at least one paper each during the 17-18 February 1966 
annual AERA convention, and another (Horn) was coauthor of a paper read there. 
Dan Stuff lebeam was on the program at the concurrent convention of NOME. 



Best regards, 

(Signed) Julian 
Julian C. Stanley 

P.8. Dick Schuts organised a five-day precession of experimental design and 
statistical analysis that was held in Chicago 12-16 February 1966. 
Instructors were Gene Glass, Les McLean, and I. 

Enclosures: Questionnaire form 
Reprint, stamped and 
addressed envelope 
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CBNTBR FOR ADVANCED STUD? IN THE BEHAVIORAL SCIENCES 
202 Jtmipero Serra Boulevard - Stanford, California 94305 

Telephone (415) 321-2052 



June 28, 1966 



SUBJECT* We greatly need questionnaires returned soon by the nine persons 
whose, as of mailtime on Monday, June 27, had not been received* 

TO : Those nine nonrespondents* 

FROM : Julian Stanley, who will soon prepare a final report on the 1965 
sumer program for the U. S. Office of Education's use in tailor- 
ing other such Institutes to the needs of educational researchers* 



Dear (As Yet) Nonrespondents: 

Won't you please take a few minutes right now and coaplete the enclosed 
questionnaire? Whether you enjoyed the Madison suonev program greatly, felt 
so-so about it, or disliked it heartily, we need to know your reactions and 
get some idea how it might have affected you during the academic year 1965-66* 
Only you can tell us that* 

Uncle Sam Invested quite a bit of money in you as consultants, with the 
understanding that you would help evaluate the program* The enclosed ques- 
tionnaire can be completed reliably in ten minutes or less by the typical 
respondent* 

We have already received completed questionnaires from twenty-one of the 
thirty participants* You nine threaten to bias the reporting in difficult- to- 
infer ways* Please don't! 



Cordially yours. 



(Signed) Julian 
Julian C* Stanley 

JC8/ap 

Enclosures: questionnaire and stamped, addressed 
envelope 
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CENTER for advanced study in the behavioral sciences 

202 Junlpero Serra Boulevard - Stanford, California 94305 

Telephone (415) 321-2052 

July 19, 1966 



Dear Three Hard-Core Nonrespondentas 

Twenty- aeven of the thirty 1965 sumner- program participant# have 
returned the questionnaire* Uncle Sam nead# your reeponaea to avoid bias! 
Please help ua repay him for supporting us during the program by completing 
the enclosed questionnaire now and returning it to me in the enclosed 
addressed, stamped envelope Immediately* 

Cordially yours, 

(Signed) Julian 

Julian C* Stanley 



JCS/ap 

Enclosures: questionnaire 

stamped, addressed envelope 
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Summary Sheet for Items Checked 

Presumed Aftereffects of the 1965 Sumner Program in Experimental Design 

Have you seemed more able this academic year, because of the program: 
to design experimental 29 Yes _1_ No Please explain: 

To analyse data resulting from experiments? 29 Yes No Please explain: 

Have you begun or completed any experiments since 13 August 1965? 

2!L Yes —14 . N*> If so, please list a short title for each: 

Have you served as a consultant on the design and analysis of experiments 
since August 13? 24 Yes 6 No If so, to whfcm and for what type of experi- 
ments? 

To what extent did the summer program seem to help you as such a consul- 
tant? ... None _ Little 7 Moderately 17 A great deal 6 No answer In 
what respects? *“ 

Have you collaborated substantially with others (either colleagues or 
students) on experiments since August 13? 15 Yes 15 No If so, to what ex- 
tent did you function as a specialist in the design and analysis of the 
experiment(s)? 

Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? 21 Yes 
- A_ No ___ 5 . NO answer If so, please explain how: 

Have you published or do you have "in press" anything affected, directly 
or indirectly, by your summer- program participation? Yes 24 No If so, 
please give the complete cltation(s) and tell how it was facilitated (jr re- 
tarded) : 

Do you have any manuscripts being considered by editors at the present time 
that were to some extent affected by your summer participation? 4 Ye s 26 No 
If so, please give the title(s) and tell how it was facilitated or inhibited by 
the summer program: 

Do you have other completed reports that owe something to your summer 
work? J9_ Yes 21 No What and how? 

Have you secured research contracts or grants subsequent to August 13? 

8 Yes 22 No If so, from what source(s)? 

For project (a) entitled 

For how many dollars each? 

Have you secured fellowships or research assistantshlps, for yourself 
and/or your students? 12 Yes 18 No If so, please explain: 

To what extent, if any, do you believe that the summer program helped you 
secure the above contracts, grants, fellowships, and/or research assistant- 
ships? None _ 5 Little 4 Moderately 5 v A great deal 12 No answer* 

Please explain: 
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Boas your attendance at professional conventions seen to have Increased, 
at least partly because of the prograaY JJ Yes 23 No 1 No answer Which 
conventions have you attended since August 137 

At which of these did you present or coauthor papers (Including symposia pre- 
sentations), and what were the titles of those papers? (X already know your 
feb* 1966 AKHA-NCME contributions * ) 

To what extent were these papers affected by your activities last sunner? 

JL JjJ tittle _ A Moderately 2 A great deal 17 No answer Please 
explains 

Ji you have taught courses with quantitative content, have you done so 
better because of the susaaer program? 12 Yes 2 No 15 Taught no such 
courses JL No response 

Other ways (salary, professional recognition, etc.) in which your achieve- 
ments were facilitated or hindered by your summer participations 

How might the summer program have been conducted differently In order to 
help you better with the design and analysis of educational experiments? 

Other comments! 

Are you scheduled to appear on future convention programs (e.g., American 
Psychological Association in New York City next September)? 4 Yes 26 NO 
Xf so, at which convention(s) and to present paper(s) entitled what? 

4 

Please estimate whether this paper or these papers resulted at least in part 
from influences of the summer program. 

Jvcrything considered, how glad are you now that you spent the eight weeks 
In Madison I*m sorry I did. 2 Neutral! neither glad nor sad. 2 It 
was worthwhile. 19 X benefited a great deal. 7 It was a hugely rewarding 
experience. Or devise your own category! 
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Predoctoral 1 

Presumed Aftereffects of the 1965 Sumner Program In Experimental Design 

Have you seemed more able this academic year, because of the program: 
To design experiments? Yes __ No, Please explain: but little oppor- 
tunity . 

To analyse data resulting from experiments? x Yes ___ No Please explain: 
X am now in a position to evaluate (rather than simply consume) re- 
search data. 



Have you begun or completed any experiments since 13 August 1965? 

Yes x No* If so, please list a short title for each: 

Have you served as a consultant on the design and analysis of experiments 
since August 13? x Yes _ ^ No If so, to whom and for what type of experi- 
ments? Assisted a Junior staff member in developing an observational instru- 
ment for classroom interaction . 

To what extent did the summer program seem to help you as such a consul- 
tant? ..... None „ Little x Moderately ^ A great deal In what respects? 
Through better understanding of significance of randomization * 

Have you collaborated substantially with others (either colleagues or 
students) on experiments since August 13? __ Yes x No If so, to what 
extent did you function as a specialist in the design and analysis of the 
experiment (s)? 

Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? x Yes 

No If so, does it seem that your eight weeks in the summer program 

helped you do this better than you would have done otherwise? x Yes __ No 
If so, please explain how: I have felt/been on much sounder ground in eval- 

uating our laboratory school program (e.g*» being more aware of self- selection 
factors « etc*) 

lave you published or do you have "in press" anything affected, directly 
or indirectly, by your summer- program participation? Yes x NO If so, 

please give the complete citation (s) and tell how it was facilitated (or re- 
tarded) : 

Do you have any manuscripts being considered by editors at the present 

time that were to some extent affected by your summer participation? Yes 

JL Mo* If so, please give the title (s) and tell how it was facilitated or 
inhibited by the summer program: 

Do you have other completed reports that owe something to your summer 
work? Yes x No What and how? 

Have you secured research contracts or grants subsequent to August 13? 

_ Yes x No If so, from what source(s)? 

For project (a) entitled 
For how many dollars each? 

Have you secured fellowships or research assistant ships, for yourself 
and/or your students? _x_ Yes __ No* If so, please explain: Extended scholar- 

ship for Ph.D* work* 
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To wh«t extent, if any, do you believe that the summer program helped you 
secure the above contracts, grante, fellowships, and/or research ass is tun t ships? 
— None Little ,_x_ Moderately A great deal. Please explain: My pro- 

jected doctoral study was more soundly baaed . 

Loss your attendance at professional conventions seem to have increased, 
at least partly because of the program? ^ Yes x No Which conventions have 
you attended since August 13? American Educational Research Association. 

LSAA . 

At which of these did you present *r coauthor papers (including symposia pre- 
sentations), and what were the titles of those papers? (1 already know your 
Feb* 1966 ABRA-MGMB contributions*) 

To what extent were these papers affected by your activities last summer? 

— . None — Little ___ Moderately A great deal Please explain: 

IS you have taught courses with quantitative content, have you done so 
better because of the summer program? Yes No x Taught no such 
courses* 

Other ways (salary, professional recognition, etc*) in which your achieve- 
ments were facilitated or hindered by your summer participation? 

How might the summer program have been conducted differently in order to 
help you better with the design and analysis of educational experiments? 

See my written evaluation submitted at the close of last atammur workshop . 

Other comments: We might, as a team* have worked on a real 'Here and now 11 

Atudy— from hypothesis and design through data collection to inferential analy- 

s is* 



Are you scheduled to appear on future convention programs (e*g*, American 
Psychological Association in New York City next September)? ___ Yes x No 
If so, at which convention(s) and to present paper(s) entitled what? 

Please estimate whether this paper or these papers resulted at least in part 
from influences of the summer program. 

Everything considered, how glad are you now that you spent the eight weeks 

In Madison? I*m sorry I did* ^ Neutral: neither glad nor sad* x It 

was worthwhile* ^ I benefitted a great deal* ^ It was a hugely rewarding 
experience. Or devise your own category: 



40- 



Predoc toral 2 



Presumed Aftereffects of the 1965 Summer Program in Experimental Design 

Have you seemed more able this academic year, because of the program: 

To design experiments? x Yes _ ^ No Please explain: 

To analyse data resulting from experiments? x Yes __ No Please explain: 

Have you begun or completed any experiments since 13 August 1965? 

X- Yes ___ No If so, please list a short title for each: Analysis of human 
behavior: children's language 

Have you served as a consultant on the design and analysis of experiments 
since August 13? x Yes __ No If so, to whom and for what type of experi- 
ments? Master's and Ph.P, students « 

To what extent did the summer program seem to help you as such a consul- 
tant? __ None Little ___ Moderately x A great deal In what respects? 

Have you collaborated substantially with others (either colleagues or 
students) on experiments since August 13? x Yes __ No If so, to what extent 
did you function as a specialist in the design and analysis of the experiment (s)? 

Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? x Yes 
__ No If so, does it seem that your eight weeks in the summer program helped 
you do this better than y*>u would have done otherwise? x Yes No If so, 
please explain how: 

Have you published or do you have '*in press' 1 anything affected, directly 
or Indirectly, by your summer-program participation? ___ Yes x No If so, 
please give the complete citation(s) and tell how it was facilitated (or re- 
tarded) 

Do you have any manuscripts being considered by editors at the present 
time that were to some extent affected by your summer participation? Yes 
x No If so, please give the title(s) and tell how it was facilitated or 
inhibited by the summer program: 

Do you have other completed reports that owe something to your summer 
work? __ Yes x No What and how? 

Have you secured research contracts or grants subsequent to August 13? 

Yes x No If so, from what source(s)? 

For project (s) entitled 
Tvr how many dollars each? 

Have you secured fellowships or research assistantships, for yourself 
and/or your students? __ Yes x No If so, please explain: 

To what extent, if any, do you believe that the summer program helped you 
secure the above contracts, grants, fellowships, and/or research assistant- 
ships? None Little Moderately A great deal* Please explain: 
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Does your attendance at professional conventions seem to have increased t 
at least partly because of the program? ^ ^ Yes x No Which conventions have 
you attended since August 13? American Educational Research Association. 
American Psychological Association » Educational Testing Service. National Coun- 
cil on Measurement in Education 

At which of these did you present or coauthor papers (Including symposia pre- 
sentations) , and what were the titles of those papers? (I already know your 
Feb. 1966 AERA-NCME contributions*) None 

To what extent were these papers affected by your activities last Sumner? 

None Little ^ Moderately ^ A great deal Please explain: 

» 

If you have taught courses with quantitative content, have you done so 
better because of the summer program? x Yes ^ No ^ Taught no such* courses. 

Other ways (salary, professional recognition, etc.) in which your achieve- 
ments were facilitated or hindered by your summer participation: Factor in my 

gaining a position this coming September. 

How might the summer program have been conducted differently in order to 
help you better with the design and analysis of educational experiments? 

Not been quite as restricted from discussions of ^psychometrics* 1 or 
non-manipiilated variable studies. 

Other comments : a 

Are you scheduled to appear on future convention programs (e.g., American 
Psychological Association in New York City next September)? __ Yes x No 
If so, at which convention (s) and to present paper (s) entitled what? 

Please estimate whether this paper or these papers resulted at least in part 
from influences of the summer program. 

Everything considered, how glad are you now that you spent the eight weeks 
in Madison? I'm sorry I did. Neutral: neither glad nor sad. ^ ^ It was 
worthwhile, x I benefited a great deal. ___ It was a hugely rewarding ex- 
perience. Or devise your own category: — 
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Presumed Aftereffects of the 1965 Summer Program In Experimental Design 

Have you seemed more able this academic year, because of the program: 

To design experiments x Yes _____ No Please explain: Have a greater famil- 
iarity with design problems and what to look for. 

To analyte data resulting from experiments? x Yes _____ No Please explain: 

Was able to acquire a greater know ledge o f t he analysis of variance, in its 
various forms. 

Have you begun or completed any experiments since 13 August 1965? x Yes 

No If so, please list a short title for each: Dissertation: Some re la* 

tionships between student 8 s socialized anxiety and their attitudes toward 
school. 



Have you served as a consultant on the design and analysis of experiments 
since August 13? x Yes _____ No If so, to whom and for what type of experi- 
ments? Students coming to the offices of research service. 

To what extend did the summer program seem to help you as such a consul- 
tant? _____ None ___ Little _____ Moderately x A great deal In what respects? 
Provided greater knowledge of both the design and analysis of experiments. 

Have you collaborated substantially with others (either colleagues or 
students) on experiments since August 13? _____ Yes x No If so, to what extent 
did you function as a specialist in the design and analysis of the experiment(e 

Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? x Yes 
, , No Zf so, does it seem that your eight weeks in the summer program helped 
you do this bet car than you would have done otherwise? x Yes _____ No If so, 
please explain how: Many considerations are the same for controlled, manipu- 

lated variable experiments and those clasified otherwise. 

Have you published or do you have ”in press” anything affected, directly 
or indirectly, by your summer- program participation? _____ Yes x No If so, 
please give the complete citation (s) and tell how it was facilitated (or re- 
tarded) : 

Do you have any manuscripts being considered by editors at the present 
time that were to some extent affected by your summer participation? _____ Yes 
x No If so, please give the title (s) and tell how it was facilitated or 
inhibited by the summer program: 

Do you have other completed reports that owe something to your summer 
work? _____ Yes x No What and how? 

Have you secured research contracts or grants subsequent to August 13? 

_____ Yes x No If so, from what source(s)? 

For project (8 > entitled 

For how many dollars each? 

Have you secured fellowships or research assistantships, for yourself 
and/or your students? ______ Yes x No If so, please explain: 
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To what extent, if any, do yon believe that the summer program helped you 
secure the above contracts, grants, fellowships, and/or research ass latent ships? 
— None —— Wfctle — Moderately _ ^ A great deal Please explain: 

®° e * your attendance at profesaional conventions seem to have increased, 

vL 1 *!!* 1 S a S tl ? becauee o£ the Program? Yes jt. No Which conventions hsvs 

attend!*^ 13? ~ Tp ^ bu8 y th;i8 year, for conventions. Will be 

At Which of these did you present or coauthor papers (including symposia pre- 
aentations), and what were the titles of those papers? (I already know your 
Feb, 1966 AERA-NCME contributions.) None 

To what extent were these papers affected by your activities last summer? 

— •*- Wone mmmm Little Moderately _ A great deal Please explain: 

If You have taught courses with quantitative content, have you done so 
better because of the summer program? Yes No x Taught no such 
courses. *~ 



Other ways (salary, professional recognition, etc.) in which your achieve- 
ments were facilitated or hindered by your summer participation: Help to some 
■S££§S£ 1 * n being offered the position of state director of research. 



How might the summer program have been conducted differently in order to 
u i? y ° U be ** :er the design and analysis of educational experiments? I 
—^l_ eVe ~th e program waswel l handled. The only "negative" c omment I mightmake 
l 8 ., -that there just wasn't e n ough time to take advantage o f all thaFwas offered 
at Madison (program wise that jail. — 

Other comments : 



_ . A y e y ou scheduled to appear on future convention programs (e.g., American 

Psychological Association in New York City next September)? Yes x No If 

ao, at which convention (s) and to present paper (s) entitled what? 

Please estimate whether this paper or these papers resulted at least in part 
from influences of the summer program. 

JjSHSyfchfofi considered, how glad are you now that you spent the eight weeks 

in Madison? I'm sorry I did. __ Neutral: neither glad nor sad. It was 

worthwhile, x I benefited a great deal. ^ It was & hugely rewarding ex-* 
perience. Or cUvise your own category: Would en joy another chance to take 

advantage of what I didn't get last summer. " 




I'm sorry this evaluation is so late, but I have been "snowed under" 
with my dissertation. 



Predoc torsi 4 
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Presumed Aftagggfgcfci of the 1965 Suggtr Program in Experimental Diiign 

Hava you seemed more abla this academic year, because of the programs 
To design experiments? *x n Tea Ho Please explain; 

To analyst data resulting from experiments? Tea Ho Please explains 

Hava you begun or conflated any experiments since 13 August 1965? 

, , ... r Tea x Ho If so, please list a short title for each; 

Hava you served as a consultant on the design and analysis of experiments 
since August 13? Tea x Ho If so. to whom and for what type of experi- 
ments? 

To what extent did the summer program seem to help you as such a consul- 
tant? _ Hone __ lilt tie __ Moderately A great deal In what respects? 

Hava you collaborated substantially with others (either colleagues or 
students) on experiments since August 13? __ Tea x Ho If so, to what ex- 
tent did you function as a specialist in the design and analysis of the 
experiment (s)? 

Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? x Yes 

Ho If so, does it seem that your eight weeks in the summer program helped 
you do this better than you would have done otherwise x Yes __ Ho If so, 
please explain hows The design is much better than it would have been without 
the program* Analysis of data also better. 

Have you published or do you have "in press" anything affected, directly 
or indirectly, by your summer- program participation? _ ^ Yes x Ho If so, 
please give the complete cltation(s) and tell how it was facilitated (or re- 
tarded) : 

Do you have any manuscripts being considered by editors at the present 
time that were to some extent affected by your summer participation __ Yea 
x Ho If so, please give the title (s) and tell how it was facilitated or 
Inhibited by the summer program: 

Do you have other completed reports that owe something to your summer 
work? Yes x Ho What and how? Dissertation in progress: Innovativeness 

in Teachers with Varying Histories. 

Have you secured research contracts or grants aubeaquert to August 13? 
__ Yes x No If so, from what source (s)? 

For project (s) entitled 

For how many dollars each? 

Have you secured fellowships or research as a 1st ant ships, for yourself 
and/or your students? ^ Yes x Ho If so, please explain: 

To what extent, if any, do you believe that the summer program helped you 
secure the above contracts, grants, fellowships, and/or research assistant- 
ships? ___ Hone __ Little ___ Moderately A great deal . 

Please explain: 








o 

ERIC 



-45- 



Does your attendance at professional convention* seem to have increased, 
at least partly because of the program? _ Yes x No Which convention* have 
you attended since August 13? In the last two years X have attended fewer , 
convention* than for the preceding ten years because of full-time study . 

At which of these did you present or coauthor paper* (including symposia pre- 
sentations), and what were the titles of those papers? (X already know your 
feb. 1966 AERA-NCMB contributions.) 

To what extent were these papers affected by your activities last summer? 

Non ® _ Little . Moderately nin .. r A great deal. Please explain: 

If you have taught courses with quantitative content, have you done so 
better because of the summer program? Yes NO x Taught no such courses , 
I have taught no courses during the past year ."""in the summer session I will 
teach a seminar in elementary school science curriculum which Is described as 
having a research orientation. Research findings will be studied with a view 
to utilisation of these findings in revision of content, methods, and sequence 
in elementary school science instruction. A graduate course in research methods 
in education is a prerequisite . 

Other ways (salary, professional recognition, etc.) in which your achieve- 
ments were facilitated or hindered by your summer participation: Appointment 
to teach the above course was certainly influenced by the attendance at the 
program. It also was given favorable consideration in salary determination for 
a 1966-67 appointment as associate professor. Duties will Include coordination 
of public school requests for college staff assistance and of college use of 
schools for laboratories. 



How might the summer program have been conducted differently in order to 
help you better with the design and analysis of educational experiments? 

More small group work with staff members would have enabled me to deal directly 
with next steps in development of my proficiency. More resource people could 
have given direct ties to other situations as did the editor . 

Other comments : The variety of backgrounds of participants made contribu- 

tions. Participants could have made more contributions to planning and instruc- 
tion . 

Are you scheduled to appear on future convention programs (e.g., American 
Psychological Association in New York City next September)? __ Yes x No If 
so, at which convention (s) and to present paper (s) entitled what? 

Please estimate whether this paper or these papers resulted at least in part 
from influences of the summer program. 

Everything considered, how glad are you now that you spent the eight weeks 
in Madison? ^ I*m sorry I did. __ Neutral: neither glad nor sad. ^ ^ It 

was worthwhile. X I benefited a~ great deal. It was a hugely rewarding 

experience. Or devise your own category: 



I 

f 
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Presumed Aftereffects of the 1965 Summer Program in Experimental Desig n 

Have you seemed more able this academic year, because of the progttm: 

To design experiments? x Yes No Please explain: 

To analyse data resulting fr<pm experiments? x Yes _ „ No Please explain: 

Have you begun or completed any experiments since 13 August 1965? 

_ ^ ei * M so, please list a short title for each: 

Have you served as a consultant on the design and analysis of experiments 
since August 13? x Yes No If so, to whom and for what type of experi- 
ments? To a number of graduate students in the Department of Psychology con- 
cerning experiments for course requirement. M.A.. and Ph.D. experiments* 

To what extent did the summer program seem to help you as such a consul- 
tant? _ None _ Little _ Moderately A great deal In what respects? 
Understanding of appropriateness of various designs, more sophisticated! , 
types of analysis. 

Have you collaborated substantially with others (either colleagues or 
students) on experiments since August 13? Yes x No If so, to what ex- 
tent did you function as a specialist in the design and analysis of the experi- 
ment (s)? 

Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? x Yes 
_ No If so, does it seem that your eight weeks in the summer program helped 
you do this better than you would have done otherwise? x Yes No If so, 
pltase explain how: Currently doing theoretical research (with computer) on 

relationships between analyses of covariance and correlational analyses, par- 
ticularly concerned with interaction effects. Summer program increased my 
understanding of mathematical basis for ANCOVAI 

Have you published or do you have "in press" anything affected, directly 
or indirectly, by your summer- program participation? __ Yes x No If so, 
please give the complete citation (si and tell how it was facilitated (or re- 
tarded) : 

Do you have any manuscripts being considered by editors at the present 
time that were to some extent affected by your summer participation? __ Yes 
x No If so, please give the title (s) and tell how it was facilitated or 
inhibited by the summer program: 

Do you have other completed reports that owe something to your summer 
work? __ _ Yes x No What and how? 

Have you secured research contracts or grants subsequent to August 13? 

__ Yes x No If so, from what source (s)? But proposal is current ly before 
Office of Education for a 5- year grant for: 

For project (a) entitled An Investigation of the Application of Same Statistical 
Techniques to Educational Research . 

For how many dollars each? About $275.000 




Hsve you secured fellowship* or research assistantships, for yourself 

and/or your student: a? _x_ Yea No If so* oleaee explains Postdoctoral 

fellow at an R & D Center. 1966-67 ! 

To what extant, if any, do you believe that the summer program helped 
you secure the above contracts, grants, fellowships, and/or research ass is* 
tantshlps? _ _ None _ Little ^ Moderately A great deal. Please ex- 

plaint 

ioes your attendance at professional conventions seem to have increased, 
at leastly partly because of the program? __ Yes _x_ No Which conventions 
have you attended since August 13? American Psychological Association, Septem- 

At Which of these did you present or coauthor papers (including symposia pre- 
sentations) , and what were the titles of those papers? (I already know your 
p€:b. 1966 AERA-NCME contributions.) Presented Programmed Instruction and the 
A bility to Learn' 1 (a correlational study, completed before the Summer Program) * 

To what extent were these papers affected by your activities last summer? 

JL None Little ____ Moderately __ A great deal Please explain: See Above 

If you have taught courses with quantitative content, have you done so 
better because of the summer program? __ Yes __ No x Taught no such 
courses. 

Other ways (salary, professional recognition, etc.) in which your achieve- 
ments were facilitated or hindered by your summer participation? 

Row might the summer program have been conducted differently in order to 
help you better with the design and analysis of educational experiments? 

Some afternoon sessions could have been Problem 11 oriented more than they ac* 
tually were, i.e. , an experiment presented with well defined independent 
variables, a general procedural outline, particularly involving problems in 
either design or analysis which required careful and creative thinking for 
their solution. 



Are you schedule to appear on future convention programs (e.g«, American 
Psychological Association in New York City next September)? — ^ Yes s No 
If so, at which convention(s) and to present paper(s) entitled what? 

Please estimate whether this paper or these papers resulted at least in part 
from influences of the summer program. 

Everything considered, how glad are you now that you spent the eight weeks 

in Madison? __ I'm sorry I did. __ Neutral: neither glad nor sad. It 

was worthwhile. x I benefited a great deal. It was a hugely rewarding 
experience. Or devise your own category. 



ii i mmmemmm ,mn m 

\ 
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Presumed Aftereffects of the 1965 Summer Program in Experimental Design 

Rave you seemed more able this academic year, because of the program: 

To design experiments? x Yes Ho Please explain: Mach more knowledge- 

able, Can critically read reports of research— much more discerning. Helped 
develop design and analysis of Title III proposal for county school system. 

To analyse data results from experiments? x Yes . No Please explain: 
Analysed data from doctoral study. Better able to cite limitations and to 
recommend tighter designs and analyses for future investigations . 

Have you begun or completed any experiments since 13 August 1965? x Yes 
No If so, please list a short title for each: Characteristics of Inno- 
vative Teachers 



Have you served as a consultant on the design and analysis of experiments 
since August 13? x Yes ___ No* If so, to whom and for what type of experi- 
ments? Served as a consultant on Title III grant to establish an independent 
study center in county schools. 

To what extent did the summer program seem to help you as such a consul- 
tant? _____ None ___ Little ___ Moderately x A great deal In what respects? 

I was much better prepared to assist in developing ways to evaluate outcomes 
for the proposal* 

Have you collaborated substantially with others (either colleagues or 
students) on experiments since August 13? _ Yes _x_ No If- so, to what extent 
did you function as a specialist in the design and analysis of the experiment (s)? 

Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? x Yes ___ 

No If so, does It seem that your eight weeks in the summer program helped you 
do this better than you would have done otherwise? x Yes ___ No If so, 
please explain how: Analyzed data from my own study of innovative teachers. 
Advised teachers in my own school who wished to conduct more quasi- type stu- 
dies of their own teaching* 

Have you published or do you have "in press" anything affected, directly 
or indirectly, by your summer- program participation? , Yes x No If so, 
please give the complete citation (s) and tell how it was facilitated (or re- 
tarded) : 

Do you have any manuscripts being considered by editors at the present 
time that were to some extent affected by your summer participation? ___ Yes 
x No If so, please give the feitle(s) and tell how it was facilitated or 
Inhibited by the summer program! 

Do yon have other completed reports that owe something to your summer 
work? ___ Yes x No What and how? 

Have you secured research contracts or grants subsequent to August 13? 

___ Yes x No If so, from what source(s)? 

For project (s) entitled 

For how many dollars each? 



Rav« you secured fellowships or research assistant ships, for yourself 
•nd/or your students? x Yes ' No If so, please explain; Two of my 
teache rs hava received ass latent shins to work In a new program In preparing 
researchers. I helped them receive the assistantahip. Mv a ssistant principal 
was appointed an Instructor In the program. 

To what extent. If any, do you believe that the summer program helped you 
secure the above contracts, grants, fellowships, and/or research asslstantshlps? 
JL None Little Moderately A great deal. Please explain: 

Does your attendance at professional conventions seem to have Increased, 
at least partly because of the program? Yes x No Which conventions have 
you attended since August 13? NASSP. Planned to attend American Educational 
Research Association, but lack of funds changed plans. 

At which of these did you present or coauthor papers (including symposia pre- 
sentations) , and what were the titles of those papers? (I already know your 
Peb. 1966 AERA-NCME contributions.) 

To what extent were these papers affected by your activities last summer? 

•JL None __ Little __ Moderately _ A great deal Please explain: 

JU yon have taught courses with quantitative content have you done so bet- 
ter because of the summer program? ^ ^ Yes No x Taught no such courses* 

Other ways (salary, professional recognition, etc.) in which your achieve- 
ments were facilitated or hindered by your summer participation: 

1. H ave served on a number of pla n ni ng committees for county schools • 

•Have r eceived an offer to direc t R/D Center for county school system about 
two years away. 

How might the summer program have been conducted differently in order to 
help you better with the design and analysis of educational experiments? 
Afternoon sessions should have been devoted* at least partially, to carrying 

through exper imental des i gns. C ould have been simulated or actual experiments* 
tion. 

Other comments: 

Are you scheduled to appear on future convention programs (e.g., American 
Psychological Association in New York City next September)? __ Yes x No 
If so, at which conventions ) and to present paper(s) entitled what? ** 

Please estimate whether this papev or these papers resulted at least in part 
from influences of the summer program. 

Everything considered, how glad are you now that you spent the eight weeks 

in MSdison? I*m sorry I did. ^ Neutral: neither glad nor sad. It 

wa«i worthwhile. x I benefited a great deal* It was a hugely rewarding 

experience. Or devise your own category: 



I am sorry that I am one of the remaining nine, but I never received 
the first questionnaire. 2 have changed residence which might account 
for the confusion. 
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Presumed Aftereffects of the 1965 Summer Program In Experimental Design 

Have you seemed more able this academic year, because of the programs 
To design experiments? jt. Yes No Please explain; 

To analyse data resulting from experiments? j Yes No Please explain: 

Hove you begun or completed any experiments since 13 August 1965? 

JSL No & 90 » plaase list a short title for each: 

Have you served as a consultant on the design and analysis of experiments 
since August 13? __x Yes No If so, to whom and for what type of experi- 
ments? Very Informal capacity 



To what extent did the summer program seem to help you as such a consul- 
tant? None Little _ x Moderately ^ A great deal In what respects? 



yon collaborated substantially with others (either colleagues or 
students) on experiments since August 13? _x_ Yes __ No If so, to what extent 
did you function as a specialist In the design and analysis of the experi- 
ment (a)? Not at all 



Have you consulted or collaborated or pursued alone research not strictly 
classifiable as controlled, variable-manipulating experimentation? x Yes 

■ in. ^ ®°* does it seem that your eight weeks in the summer program helped 

you do this better than you would have done otherwise? Mayb e Yes x No If 

so, please explain how; Am working on a factor-ana lytic paper, something I've 
never dabbled with before* '' 1 

Have you published or do you have "in press" anything affected, directly 
or indirectly, by your summer- program participation? Yes _x_ No If so, 

please give the complete citatlon(s) and tell how it was facilited (or retar- 
ded): 



Do you have any manuscripts being considered by editors at the present 

time that were to some extent affected by your summer participation? Yes 

•JSL* N° If so, please give the title (s) and tell how it was facilitated or 
inhibited by the summer program: 

Do you have other completed reports that owe something to your summer 
work? Yes x No What and how? 



Have you secured research contracts or grants subsequent to August 13? 
Yes x No If so, from what source(s)? 

For project (s) entitled 

For how many dollars each? Although my answer is no, I oblect to this cues 
tlon! 



Have you secured fellowships or research assistant ships, for yourself 
and/or your students? Yes _x_ No If so, please explain? 



